The field of remote control has come a long way since the days of watching a model aircraft fly under the control of a handheld controller. Robotics and remote robotic manipulation have created a strong and pressing need for more remote and better remote control systems. Obviously, an ideal form of remote control involves providing an operator with all the sensations of operating the remote robot without the inherent dangers, travel, and so forth. In order to achieve this, a telepresence system is used.
Telepresence systems are sensory feedback systems for allowing sensing and monitoring of remote systems. A typical telepresence sensor is a camera and a head mounted display. The system provides visual feedback from a remote location to an operator. For example, in a telepresence system for an automobile, the front windshield is provided with a camera. The controls of the vehicle are provided with actuators for automatically manipulating same. An operator is provided with a duplicate of the cabin of the car. The windshield is replaced with a display and the controls are linked via communications to the actuators within the vehicle. Turning of the steering wheel in the cabin of the car causes the steering wheel to turn in the vehicle. Similarly, the camera captures images in front of the car and they are displayed on the display in the cabin of the car.
Presently, there is a trend toward providing the visual feedback using a head mounted display (HMD). A head mounted display is a small display or two small displays mounted for being worn on a users head. Advantageously, an HMD with two displays provides stereo imaging allowing a user to perceive depth of field. Alternatively, such an HMD provides two identical images, one to each display. Unfortunately, the head mounted display only presents a user with information from approximately in front of the user. Thus, when a user turns their head, the image seen and the expected image differ. Therefore, the camera is mounted on a mechanism which moves in accordance with detected HMD movement. Thus, the image before the user is in accordance with the user's head position.
Generally, it is an object of telepresence systems to provide a visual sensation of being in the place of the robot and a control system for controlling the robot as well. Thus, telepresence systems aim to provide feedback that is appropriate to different situations.
Unfortunately, a camera does not move in exact synchronisation with the HMD so the image is not perfectly aligned with the expectations of the user during head motion. This misalignment can result in disorientation and nausea on the part of an operator.
The disclosures in U.S. Pat. No. 5,579,026 issued on Nov. 26, 1996 in the name of Tabata and in U.S. Pat. No. 5,917,460 issued on Jun. 29, 1999 in the name of Kodama focus on image display for use in, for example, virtual reality and games. In there is described a head mounted display in which the position of the projected image can be displaced in response to a control unit or in response to the rotational motion of the operator's head. The essence of the head-tracking implementation is that from the user's perspective, the image can be made to remain substantially stationary in space during head movements, by being manipulated in a manner opposite to the movements. Significantly, the patents do not relate to visual telepresence using slaved cameras. In the slaved camera implementation, the camera should follow the motion of the HMD and, as such, compensation for HMD motion is unnecessary since the image is always of a direction in which the head is directed.
Further because U.S. Pat. No. 5,579,026 relates to displaying a simulated planar image, such as a simulation of a television screen located in virtual space in front of the user, the patent provides for a fixed frame of reference relative to a wearer of the HMD. The images in any direction are simulated thus being formed as needed. Unfortunately, in telepresence systems, often the video data relating to a particular direction of view is unavailable. This complicates the system significantly and as such, the prior art relating to video data display is not truly applicable and, one of skill in the art would not refer to such.
In U.S. Pat. No. 5,917,460 issued on Jun. 29, 1999 in the name of Kodama a system addressing the three-axes displacement (up/down, left/right, frontwards/backwards) of a HMD is provided. The displacement appears to linear and is accommodated through a mechanical mechanism. The displays are moved in response to detected movement of a head and as such, objects remain somewhat stationary from the visual perspective of the user.
It is not well suited to use in telepresence wherein a camera tracks the motion of the HMD. One of skill in the art, absent hindsight, would not be drawn to maintaining a visual reference when a head is turned, for a telepresence system wherein a camera is rotated in response to head movement. Of course, the different problem results in a different solution.
For example, in telepresence systems, the delay between camera image capture and head motion is often indeterminate. It is not a workable solution to implement the system of the above referenced patents to solve this problem. Because of the unknown delays caused by camera response time and communication delays, the solution is not trivial.
In U.S. Pat. No. 5,933,125 a system is disclosed using prediction of the head movement to pre-compensate for the delay expected in the generation of a virtual image, nominally in a simulated environment. By this means, a time lag in the generation of imagery is compensated for by shifting the scene to provide a stable visual frame of reference. This method is applicable to short delays and small displacements, where head tracking information can be used to predict the next head position with reasonable accuracy. The patent discloses 100 msec as a normal value. Effective prediction of head motion is aided by comprehensive information about head movement, including angular head velocity and angular acceleration. For small head movements, errors induced are small. Typically, these occur in a small period of time. The disclosed embodiments rely on knowledge of the time delay, which is nominally considered to be constant. Unfortunately, when the time delays grow large allowing for substantial motion of a head, the errors in the predictive algorithm are unknown and the system is somewhat unworkable.
Furthermore, U.S. Pat. No. 5,933,125 cannot compensate for unanticipated image movement, only that which occurs in correct response to the operator's head movement. Also, it does not relate to visual telepresence systems using remote slave cameras.
It would be highly advantageous to provide a system that does not rely on any form of prediction for compensation and which works with variable delays between image capture and image display.